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PREFACE 


The  Air  Force  Research  Laboratory's  (AFRL's)  Value  of  Information  in  Collaborative  Systems 
(VICS)  initiative  is  developing  technology  to  enable  collaborative  control  of  a  flexibly 
autonomous  system.  The  objective  of  this  research  is  to  investigate  the  effects  differing 
communication  levels  and  automation  display  concepts  have  on  supervisory  control  of  multiple 
UAVs,  (711  HPW/RHCI),  Work  Unit  53290901,  Supervisory  Presentation  for  Research, 
Information  Integration  &  Testing  (SPRINT). 


1.0 


SUMMARY 


As  capabilities  of  autonomous  systems  expand,  traditional  geospatial  information  displays  used 
for  supervisory  control  will  likely  need  to  be  augmented  with  more  explicit  information  on 
higher-order  autonomous  activities,  such  as  goal-directed  task  selection,  situation  assessment, 
decision-making,  and  planning.  We  present  a  supervisory  control  interface  for  higher-order 
autonomy  based  on  finite  state  machine  diagrams  called  Layered  Pattern  Recognizable  Interfaces 
for  State  Machines  (L-PRISM).  L-PRISM  is  a  hierarchically  arranged  set  of  nested  state 
diagrams  coupled  with  a  temporal  control  and  payload  viewer.  The  mission  goals  and  tasks  are 
at  the  top-most  layer  and  sub-tasks  and  states  at  lower  layers,  providing  varying  levels  of 
information  abstraction.  Task  diagrams  use  unique  layouts  to  facilitate  pattern  recognition, 
which  is  anticipated  to  improve  the  operator’s  situation  assessment.  Furthermore,  system 
behaviors  can  be  viewed  in  real  time  or  retrospectively  evaluated.  This  paper  will  describe  the 
L-PRISM  concept  and  plans  for  examining  its  effect  on  supervisory  control  performance. 


2.0  INTRODUCTION 

The  Air  Force  Research  Laboratory’s  (AFRL’s)  Value  of  Information  in  Collaborative  Systems 
(VICS)  initiative  is  developing  technology  to  enable  collaborative  control  of  a  flexibly 
autonomous  system.  The  system  is  characterized  as  being  composable  and  fractionated  (United 
States  Air  Force  Chief  Scientist,  2010)  where  heterogeneous  members,  both  human  warfighters 
and  unmanned  systems,  within  the  system  can  assemble  and  combine  their  respective  capabilities 
and  strengths  to  achieve  mission  objectives.  The  heterogeneous  components,  under  varied 
communication  conditions,  can  coordinate  with  each  other  and  have  the  latitude  to  perform 
decentralized  planning  and  decision-making  as  the  situation  dictates.  The  goal  is  to  enable  agile 
and  adaptive  mission  management  and  control  for  a  team  comprised  of  unmanned  aerial  vehicles 
(UAVs),  unattended  ground  sensors  (UGS),  dismounted  warfighters  with  mobile  control  stations, 
and  an  operator  located  in  a  central  control  station.  With  UAVs  equipped  and  authorized  to  re¬ 
plan  and  act  without  human  input  and  under  complex,  uncertain  conditions,  a  significant 
technical  challenge  being  addressed  is  enabling  effective  supervisory  control  of  the  autonomous 
system.  Specifically,  the  challenge  is  developing  methods  for  a  human  operator  to  sufficiently 
monitor,  inspect,  and  manipulate  the  UAVs’  activities,  which  include  goal-directed  task 
selection,  situation  assessment,  decision-making,  planning,  and  actions. 

Much  of  the  past  emphasis  in  developing  multi-UAV  supervisory  control  interface  technology 
has  been  on  effectively  portraying  the  geo-spatial  (e.g.,  a  tactical  situation  map)  and  vehicle 
status  (e.g.,  flight  parameters,  navigation  mode,  fuel  level)  information.  The  associated  displays 
(see  example  of  multi-UAV  control  station  in  Figure  1)  have  transitioned  well  for  many 
supervisory  control  concepts  as  these  have  typically  been  applications  where  the  mission 
planning  and  management  was  done  with  the  operator’s  direct  and  near-continuous  involvement 
(Feitshans,  Rowe,  Davis,  Holland,  &  Berger,  2008;  Patzek  et  al.,  2009).  However,  as  future, 
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more  autonomous  systems  assess  situations  and  create  their  (re-)plans  with  little  or  no  human 
involvement,  the  information  typically  contained  in  the  tactical  situation  map  and  system  status 
displays  will  not  be  adequate  for  the  operator  to  fully  understand  what  the  vehicles  are  doing  and 
why  in  every  case.  This,  combined  with  periodic  (versus  continuous)  updates  on  vehicle 
locations,  status  and  plans,  may  lead  to  significant  operator  uncertainty  and  errors  in  supervisory 
judgment. 


Figure  1.  AFRL’s  Vigilant  Spirit  Control  Station:  An  Example  of  a  Multi-UAV  Control 
Station.  A)  Tactical  Situation  Map,  B)  Vehicle  Summary  Information,  C)  Vehicle  Management 
Panel,  and  D)  Sensor  Management. 


Given  well-known  human-automation  interaction  shortcomings  (e.g.,  mode  confusion,  vigilance 
decrement,  lack  of  appropriate  feedback,  automation  complacency,  and  bias)  and  foreseeing 
supervisory  control  challenges  with  a  dynamic  multi-vehicle  autonomous  system,  we  are 
investigating  supervisory  control  interface  concepts  to  display  higher-order  mission  management 
and  control  information  pertaining  to  the  vehicles’  progress  towards  mission  goals,  tasks,  and  the 
underlying  rationale  for  plan  changes  during  mission  execution.  The  goals,  tasks,  and  rationale 
are  intended  to  provide  a  window  into  the  vehicle’s  perception  and  assessment  of  the  situation. 

In  addition  to  providing  more  detailed  information  concerning  the  information  processing  and 
behavior  of  the  autonomous  system,  there  is  a  challenge  to  present  the  information  in  a  manner 
that  affords  quick  and  accurate  assessment  of  the  multi-vehicle  system.  With  this  in  mind,  our 
goal  is  to  design  concepts  that  can  support  the  use  of  symbols  and  patterns  in  an  attempt  to 
support  “at  a  glance”  recognition  of  complex  activities. 

We  examined  design  visualization  techniques  used  to  illustrate  and  represent  complex,  multi¬ 
state  systems  and  processes  with  the  idea  that  perhaps  these  methods  could  be  made  dynamic  and 
animated  to  represent  real-time  (and/or  past)  activities.  Diagram  and  graph  methods  such  as 
flowcharts,  binary  decision  trees,  goal  graphs,  finite  state  machine  diagrams,  and  petri  nets  were 
assessed  for  their  potential  application  for  an  intuitive  dynamic  display  of  the  autonomous 
system.  Finite  state  machine  diagrams  were  selected  given  their  efficiency  to  display  multiple 
systems  traversing  multiple  tasks  and  states,  and  the  ability  to  nest  states  (Harel,  1987)  within 
tasks  to  represent  rules,  constraints  and  overall  mission  decision  logic.  Furthermore,  the  arcs  or 
directional  lines  between  states  represent  the  conditions  that  must  be  met  by  the  systems  to  be 
able  to  move  from  one  state  into  the  next,  which  supports  the  notion  of  providing  the  operator  the 
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rationale  for  autonomous  mission,  task,  and  state  changes.  Another  reason  for  choosing  finite 
state  machine  diagrams  as  the  basis  for  the  interface  was  that  they  can  be  formed  in  a  manner  to 
produce  unique  layouts  or  patterns,  supporting  the  goal  to  develop  an  interface  that  fosters 
efficient  recognition  of  the  activities.  Finally,  a  form  of  a  state  diagram  user  interface  has  been 
demonstrated  for  robot  mission  planning  and  representing  robot  tasks  (MacKenzie,  Arkin,  & 
Cameron,  1997;  Endo,  MacKenzie,  &  Arkin,  2004). 

Currently,  we  are  developing  hierarchical  pattern-oriented  state  diagram  concepts  to  represent 
the  autonomous  activities.  In  addition,  a  control  timeline  and  payload  viewer  are  also  being 
developed  to  navigate  and  inspect  the  past  events  and  associated  details,  including  images, 
videos,  audio  recordings,  and  text  messages.  The  layered  finite  state  machine  diagrams 
combined  with  a  control  timeline  and  payload  inspection  display,  collectively  referred  to  as 
Layered  Pattern  Recognizable  Interfaces  for  State  Machines  (L-PRISM),  will  be  integrated  with 
the  multi -UAV  control  station’s  tactical  situation  map  and  system  status  information  to  assist  the 
operator  to  not  only  be  aware  of  vehicles’  locations  and  planned  routes  but  also  their  mission 
goals,  associated  tasks,  and  states  to  achieve  the  mission  goals.  The  remainder  of  this  paper  will 
describe  the  L-PRISM  concept  and  plans  for  its  testing  and  evaluation. 


3.0  THE  L-PRISM  CONCEPT 

The  L-PRISM  concept,  shown  in  Figure  2,  is  composed  of  three  display  components:  A)  the 
State  Diagram,  B)  the  Timeline  and  C)  the  Payload  Viewer.  The  State  Diagram  uses  the 
conventions  of  finite  state  machine  diagrams  to  represent  the  autonomous  system  activity  to  the 
operator  in  real  time.  The  State  Diagram  depicts  the  autonomy  through  the  state  nodes  and 
transition  arcs  of  the  diagram.  State  nodes  are  display  elements  that  represent  autonomous  tasks, 
showing  what  the  vehicle  is  doing.  The  transition  arcs  represent  the  specific  criteria  for  changing 
tasks,  showing  why  the  vehicle  may  move  to  a  new  task.  Figure  2  shows  the  diagram  for  the 
“Monitor”  task,  which  has  five  sub-task  or  state  nodes:  Patrol,  Isolate,  Capture,  Deliver,  and 
Response.  When  a  task  change  occurs,  the  transition  arc  is  animated  with  the  vehicle  color  to 
emphasize  the  task  change  and  help  mitigate  change  blindness  (see  Simons,  2000,  for  a  review). 

Expanding  on  the  conventions  of  finite  state  machine  diagrams,  multiple  vehicles  can  be 
displayed  in  one  diagram  to  accommodate  multi-UAV  monitoring  and  control.  The  State 
Diagram  simultaneously  shows  the  state  of  each  UAV  in  the  mission  through  vehicle  icons. 
Vehicle  icons  depict  the  vehicle  type,  identify  the  vehicle  by  its  call  sign  and  unique  color,  and 
provide  a  time-on-task  clock.  The  UAV’s  call  sign  and  unique  color  are  consistent  throughout 
the  rest  of  the  control  station  and  support  visual  momentum  when  transitioning  between  different 
displays.  The  time-on-task  clock  shows  how  long  the  vehicle  has  been  working  in  a  specific  task. 
If  a  vehicle’s  task  changes,  its  icon  moves  to  a  new  task.  The  autonomous  system  can  only  make 
task  and  sub-task  changes  that  follow  the  transition  arcs.  For  example,  in  the  Monitor  task, 

Patrol  can  only  lead  to  Isolate.  The  operator,  however,  is  able  to  reassign  vehicles  to  any  task  or 
sub-task  regardless  of  transition  arcs  by  dragging  and  dropping  the  UAV  symbol  into  the  desired 
task  (i.e.,  direct  manipulation).  For  tasks  that  have  no  transition  arcs,  such  as  the  middle  task  in 
Figure  2’s  State  Diagram,  task  changes  can  only  be  made  by  operators.  In  general,  vehicle 
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control  within  L-PRISM  has  the  flexibility  to  support  different  levels  of  automation  (e.g., 
management  by  exception,  management  by  consent,  operator  directed),  as  long  as  there  is 
adequate  communication  with  the  particular  vehicle(s). 


Figure  2.  L-PRISM  and  its  three  components:  A)  the  State  Diagram,  B)  the  Timeline,  and  C) 
the  Payload  Viewer. 


Another  expansion  on  the  conventions  of  finite  state  machine  diagrams  is  in  how  L-PRISM  uses 
a  layered  arrangement  of  nested  state  diagrams  to  provide  representation  of  autonomous  tasks  at 
varying  levels  of  abstraction.  L-PRISM’ s  State  Diagram  is  currently  made  up  of  three  levels  or 
layers:  the  Mission  Layer,  the  Task  Layer,  and  the  Sub-Task  Layer  (Figure  3).  These  levels  of 
abstraction  are  expected  to  enhance  understanding  and  management  of  autonomous  activities  in 
part  by  displaying  connections  between  actions,  plans,  and  goals.  The  Monitor  task  diagram 
(described  above  and  shown  in  Figure  2)  is  an  example  of  a  Task  Layer  that  shows  sub-task 
components  making  up  the  monitoring  mission  task.  At  a  more  abstract  level,  the  State  Diagram 
shows  the  Mission  Layer  (see  Figure  3)  where  each  mission  task  (e.g.,  Monitor,  Surveillance, 
Overwatch)  contains  its  sub-tasks.  Active  tasks  and  sub-tasks  are  identified  with  UAV  icons 
shown  in  them.  Additional  details  concerning  the  tasks  and  the  UAVs  performing  the  tasks  can 
be  found  by  viewing  the  associated  Task  Layer.  To  maintain  overall  system  awareness  when 
accessing  a  Task  or  Sub-Task  layer,  a  small  higher  level  state  diagram  is  portrayed  in  the  top  left 
comer  of  the  display.  It  shows  the  UAVs’  mission  task  assignments  and  also  functions  as 
method  to  navigate  back  to  a  higher  layer.  Across  all  levels,  each  state  machine  has  a  uniquely 
patterned  layout  to  redundantly  code  the  individual  tasks,  attempting  to  facilitate  “at  a  glance” 
recognition.  The  Monitor  task  layer  can  be  identified  by  its  unique  “X”  layout,  which  is  viewable 
at  abstract  and  detailed  levels. 
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Sub-Task  Layer 


Figure  3.  L-PRISM’s  layers  of  nested  state  diagrams  showing  varying  levels  of  abstraction. 
The  unique  patterns  of  state  diagrams  are  persevered  across  different  layers.  At  the  Task  and 
Sub-Task  layer,  a  small  high-level  view  is  presented  to  maintain  some  overall  system  awareness. 


Lastly,  the  State  Diagram  can  be  used  to  build  or  edit  any  layer  (Mission,  Task,  or  Sub-Task). 
When  in  “Edit  mode,”  a  tool  bar  appears  that  allows  new  state  nodes  and  new  transition  arcs  to 
be  added  to  the  task  diagram.  Additionally,  all  the  current  state  nodes  and  transition  arcs  can  be 
moved  or  deleted.  Any  changes  are  snapped  to  a  grid,  to  facilitate  the  layout  of  new  patterns. 
Every  node  and  transition  arc  has  a  label  that  can  be  edited  as  well. 

The  Timeline  (see  Figure  2)  presents  information  on  significant  mission  events  and  provides 
controls  to  retrospectively  review  that  information  across  the  control  station.  An  event  is 
represented  by  a  colored  tile  containing  a  letter  or  icon  that  depicts  a  vehicle  task  change  or 
payload  delivery.  Colors  match  the  vehicle  color  that  is  used  throughout  the  control  station,  and 
icons  show  the  type  of  payloads.  When  used  as  a  controller,  the  time  selected  on  the  Timeline 
updates  the  State  Diagram,  the  Payload  Viewer,  tactical  situation  map  and  vehicle  summary 
displays  with  the  corresponding  information  at  that  time.  Operators  can  proceed  forward  or 
backward  in  time  to  investigate  the  tasks  and  actions  of  the  vehicles  and  rationale  behind  those 
actions  using  controls  for  temporal  navigation:  zoom  the  time  scale  in  and  out,  scroll  forward  and 
back,  step  serially  through  events,  and  return  to  current  time.  This  retrospective  functionality  is 
in  contrast  to  many  timeline  applications,  which  represent  what  is  planned  or  scheduled  to 
happen.  This  retrospective  was  designed  to  gain  better  situation  awareness  of  past  activities  and 
help  in  supervisory  control  of  the  overall  system  given  challenges  associated  with  periodic 
communication  conditions  (e.g.,  decentralized  control).  For  a  vehicle  that  regained 
communication  with  the  control  station  after  being  out  of  communication  for  a  period  of  time,  all 

5 


Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88ABW  Cleared  2/15/2013;  88ABW-20 13-0861. 


of  its  activities  will  populate  the  Timeline  and  are  distinguished  from  other  entries  by  a  dashed 
border.  In  addition  to  the  retrospective  functionality,  we  plan  to  expand  the  timeline  to  include  a 
view  of  autonomous  system’s  scheduled  tasks  and  develop  a  means  to  contrast  the  plan  with 
present  and  past  activities.  Finally,  the  Timeline  is  integrated  with  the  Payload  Viewer,  such  that 
selection  of  a  payload  event  on  the  Timeline  will  highlight  and  display  that  payload  in  the 
Payload  Viewer. 

The  Payload  Viewer  (see  Figure  2)  displays  a  sortable  list  of  mission-relevant  data  referred  to  as 
payloads.  Typical  payloads  are  images  or  videos  from  UAV  sensors,  but  could  be  operator¬ 
generated  images  or  videos  as  well.  Additionally,  all  operators  (i.e.,  dismounted  warfighters 
with  mobile  control  stations  and  the  stationary  central  control  station)  can  create  and  send  text 
messages  or  “voice  note”  audio  recordings  as  payloads.  Because  many  of  the  payloads  are  an 
asynchronous  form  of  communication  between  the  system  components,  the  list  of  payloads  was 
designed  to  be  roughly  analogous  to  email.  Each  line  entry  in  the  list  corresponds  to  a  single 
message  that  can  have  multiple  payloads  attached.  The  messages  and  individual  payloads  show 
who  created  the  message,  what  time  it  was  created,  the  payload  type  icon,  and  the  message 
priority.  The  list  of  payloads  allows  sorting  and  filtering,  so  that  operators  can  quickly  find  any 
payload.  When  selected,  payloads  are  displayed  in  the  top  window  with  the  appropriate  controls 
(e.g.,  a  video  payload  uses  video  playback  controls). 


4.0  FUTURE  PLANS 

The  L-PRISM  concept  has  evolved  into  a  functional  design  and  plans  are  underway  for  an 
empirical  study  to  assess  the  effect  L-PRISM  has  on  an  operator’s  supervisory  control 
performance.  The  plan  is  to  conduct  a  set  of  complex,  multi-vehicle  mission  simulations  using 
AFRL’s  Vigilant  Spirit  Control  Station  with  and  without  L-PRISM.  The  mission  context  will  be 
reconnaissance,  surveillance,  and  target  acquisition  related  to  finding  a  suspect  vehicle  within  a 
large  road  network,  referred  to  as  “road  monitoring”.  The  UAVs  will  be  operating,  for  the  most 
part,  in  a  decentralized  control  manner  where  they  will  patrol  and  isolate  the  location  of  the 
suspect  vehicle  in  collaboration  with  the  unattended  ground  sensors  and  dismounted  warfighters. 
L-PRISM  will  depict  the  vehicles’  tasks  and  task  changes  to  display  the  adaptive  autonomous 
systems.  The  emphasis  will  be  on  assessing  L-PRISM’s  impact  on  the  operator’s  situation 
awareness  and  the  ability  to  detect  an  anomaly  or  fault  (e.g.,  an  incorrect  task  or  action).  Mental 
workload  and  confidence  ratings  will  also  be  compared  across  the  conditions. 

Two  additional  independent  variables  are  being  considered  for  the  first  or  perhaps  a  subsequent 
study.  The  first  would  vary  communication  between  the  heterogeneous  system  members  from 
continuous,  real-time  communication  to  periodic.  In  past  evaluations  under  continuous 
communication,  operators  would,  on  occasion,  attempt  to  infer  the  automation  activities  from  the 
tactical  situation  map.  We  are  interested  in  examining  if  the  L-PRISM  information  changes  the 
reliance  on  the  map  for  assessing  the  vehicles’  tasks  and  how  the  communication  level  affects 
the  operator’s  behavior.  The  second  variable  of  interest  is  the  reliability  of  the  autonomous 
planning  and  actions  and  its  effect  on  the  use  of  L-PRISM  information  given  the  operator’s 
understanding  of  the  reliability  and  resulting  trust  levels.  Here  the  hypothesis  is  that  the  L- 
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PRISM  display  would  be  relied  on  more  for  autonomous  systems  prone  to  more  situation 
assessment  and  planning  mistakes  as  L-PRISM  provides  a  more  detailed  medium  to  inspect  the 
higher-order  activities. 

As  part  of  the  VICS  initiative,  the  Vigilant  Spirit  Control  Station  and  L-PRISM  design  will  be 
integrated  with  decentralized  control  algorithms  for  a  multi-UAV  flight  test  to  demonstrate  the 
road  monitoring  mission  task.  The  plan  is  to  place  unattended  ground  sensors  along  a  large  road 
network  and  have  the  UAVs  collect  information  from  them  and  locate  vehicles  detected  by  the 
UGSs  or  dismounted  warfighters.  The  L-PRISM  information  will  be  available  to  both  the 
central  control  station  and  the  mobile  control  stations  used  by  the  dismounted  warfighters.  L- 
PRISM  is  expected  to  provide  critical  information  to  understand  the  complex  conditions  and 
activities  associated  with  this  composable  and  fractionated  autonomous  system. 


5.0  SUMMARY 

In  summary,  L-PRISM  is  an  evolving  supervisory  control  display  concept  to  enable  an  operator 
to  quickly  gain  situation  awareness  and  effective  oversight  of  adaptive  autonomy  and  the 
associated  systems:  their  mission  goals,  tasks,  states,  and  the  underlying  rationale  for  adaptive 
plans  and  actions.  L-PRISM  shows  promise  for  providing  many  of  the  desired  attributes  and 
features  for  displaying  information  on  a  highly  autonomous  multi-vehicle  system.  It  supports 
scaling  of  the  team  size,  the  heterogeneous  systems  and  capabilities,  assorted  mission  tasks  and 
embedded  states,  both  real-time  and  periodic  communication  conditions,  the  patterns  for 
recognizing  the  tasks,  states  and  overall  situation,  and  can  and  will  be  integrated  within  an 
existing  multi-vehicle  supervisory  control  station. 
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